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(54) Zoomorphic computer user interface 

(57) A zoomorphic computer for controlling a com- 
puter system includes an animal or humanoid shaped 
shell (10) having an attached transceiver (32) for two 
way communication with a computer system. A position 
detecting unit (22) determines position of the zoomor- 
phic shell relative to the computer system, with change 
of position of the zoomorphic shell relative to the com- 
puter system changing state the zoomorphic shell or the 
computer system. The zoomorphic shell (10) can have 
movable elements such as arms or tails, attached to the 
zoomorphic shell, and support a feedback unit that com- 
municates with the computer system, modifying position 
of a movable element in response to computer system 
output. 
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Description 

[0001] The present invention relates to a zoomorphic 
computer user interface. 

[0002] Computer input and output is predominantly 
supported by keyboards and mice for input, along with 
cathode ray tube (CRT) monitors, flat panel displays, 
and audio speakers for output. While such input/output 
devices are both powerful and widely available, they do 
have limitations. For example, novice computer users 
are often uneasy in the presence of "high tech" input/ 
output devices, while even skilled computer users may 
avoid types of computer interactions because of the per- 
ceived effort required to accomplish a task. 
[0003] In accordance with a first aspect of the present 
invention, a zoomorphic computer user interface com- 
prises 

a computer system directable both to accept user 
input to modify its system state, and to provide user 
output, 

a zoomorphic unit connected to the computer sys- 
tem to provide user input and react to computer sys- 
tem output, 

a reference unit connected to the computer system 
and available lor interaction with the zoomorphic 
unit, with positioning of the zoomorphic unit with re- 
spect to the reference unit directably modifying sys- 
tem state of the computer system. 

[0004] To overcome adverse perceptions and facili- 
tate common computer interactions such as logging on, 
providing passwords, or enabling communication appli- 
cations, the present invention therefore contemplates 
the use of small auxiliary computers separate and dis- 
tinct from a user's primary computer. To enhance ap- 
peal, such "tangible" computers can be constructed to 
simulate conventional objects having widespread user 
recognition. For example, a small, handheld computer 
used to provide passwords to various applications can 
be shaped as a stylized key. A user can rotate the key 
in front of a computer monitor, with internal rotation sen- 
sors in the key triggering a wireless sending of a logon 
and password sequence to 'unlock" a computer or com- 
puter application. 

[0005] The present invention envisages a computer 
processor and display having animal, human, or object 
shaped input/output devices tor visual, auditory, grasp- 
ing, gripping, or touching mediated user input. 
[0006] One particularly useful form of tangible com- 
puter is configured to have an animal or human shape. 
This would include real or fantastic animals (horses or 
unicorns), doll shapes or action figures, or even imagi- 
nary monsters or aliens. Such zoomorphic computers 
are readily accepted by younger computer users, and 
easily recognizable by any computer user. Advanta- 
geously, a rich symbolic or metaphoric vocabulary is 
available to guide use of such zoomorphic tangible com- 



puters. For example, a growling dog figure can be rec- 
ognizable as a "guard dog", and may provide a suitable 
substitute to the stylized key tangible computer for pass- 
word protection of a main computer. 
5 [0007] As those skilled in the art will appreciate, reli- 
ably, quickly, and intuitively transmitting complex com- 
mands to small tangible or zoomorphic computers can 
be difficult. Small computer devices do not generally 
have sufficient computer processing power to respond 
10 reliably to voice or handwritten (pen based) commands, 
keyboards are often absent or too small for accurate fin- 
ger input, and conventional buttons are too large or sup- 
port too limited a command instruction set. User inter- 
face techniques that rely on bulky external modules (full 
is size infrared linked keyboards, tethered data gloves, or 
camera based gestural recognition equipment, for ex- 
ample) are expensive, often not readily available out- 
side selected sites, and probably too awkward for wide- 
spread use in conjunction with consumer level portable 
computing devices. 

[0008] User interface designers for traditional, non- 
zoomorphic portable computers have attempted to com- 
pensate for some of these problems by constructing de- 
vices that rely on various spatial, positional, or environ- 
mental cues that manually or automatically allow for ac- 
tivation of various modes in the device. For example, 
some laptop computers use the action of opening/clos- 
ing the lid to initiate automatic bootup/powerdown of the 
computer without requiring any additional signal input 
(e.g. such as depressing a "start" button or typing 
M Lo_g_o_n n on a keyboard) from a user. Alternatively, 
the use of small portable computers that automatically 
switch control modes depending on position, orienta- 
tion, or room location have been investigated. Button - 
less manual control of a portable computer through de- 
liberate user actions such as tilting the portable compu- 
ter have also been described. 

[0009] However, all these solutions for interfacing with 
small portable computers have generally been limited 
in scope and functionality. What is needed is a user in- 
terface system suitable for even very small portable 
computers (having volumetric dimensions on the order 
of a one cubic centimeter) that is powerful, can be intu- 
itively operated by an ordinary user with little training, 
and is still readily capable of modification or extension 
by the user. The present invention meets these require- 
ments by providing a manipulatory user interface that 
responds to a user twisting, folding, bending, squeezing, 
shaking, tilting, spinning, lifting, or otherwise physically 
manipulating the zoomorphic computer. 
[0010] In the manipulatory user interface system of 
the present invention, the most basic level of manipula- 
tion is known as a "senseme". A senseme is defined as 
a single indivisible type of physical manipulation. A par- 
tial list of categories of sensemes include material trans- 
formations such as squeezing, twisting, stretching; local 
spatial transformations such as translation, rotation, or- 
biting; and environmental transformations based on 
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temperature, light level, or vibration. For example, a 
small portable computer may support a deformable 
piece having multiple embedded sensors that detect 
folding, twisting, or bending of the deformable piece by 
a user. This computer can also contain a number of ac- 
celerometers that sense relative spatial information; gy- 
roscopic, radio or infrared positional sensors for deter- 
mining absolute position; and various thermal or photo- 
sensors that respectively detect temperature and light 
level changes. Intentional or unintentional modifications 
detected by one or more of these sensor systems can 
provide the basis for a powerful user interface scheme. 
[0011] As will be appreciated, each senseme catego- 
ry contains many individually distinguishable members. 
For example, consider the category of senseme known 
as a "pinch 0 , a structural transformation generally com- 
pleted by a user squeezing the deformable piece be- 
tween a forefinger and thumb. A pinch can be modified 
by varying its speed (quick or slow pinch), magnitude/ 
intensity (light or hard pinch), portion of deformable 
piece pinched (top, bottom, or center of deformable 
piece pinched), or even portion of body used to pinch 
(right handed pinch or left handed pinch), with each 
modification being distinguishable as a senseme capa- 
ble of being mapped onto a computer control command. 
[001 2] Although the wide variety of easily distinguish- 
able sensemes would alone provide a powerful user in- 
terface to a computer, a preferred embodiment of the 
present invention further extends the flexibility of the 
senseme based user interface by supporting computer 
control based on a "morpheme" input. The morpheme 
is a temporally synchronous (or overlapping asynchro- 
nous) tuple of one or more sensemes. Note that a mor- 
pheme can (and often will) contain more than one 
senseme. The sensemes combined into a morpheme 
can come either from the same category (the user pinch- 
es with a right hand while tapping with a left hand finger), 
or different categories (the user pinches the deformable 
piece with a right hand while modifying the spatial posi- 
tion of the portable computer by tilting it forward). 
[001 3] Any morpheme can in turn be extended by par- 
ticipation in a "sentence". A sentence is defined as a 
sequence of one or more temporally disjoint mor- 
phemes. The sentence level allows definition of a phys- 
ical manipulatory grammar by appropriate choice of 
morpheme sequence, and corollary rules governing, for 
example, use of active (verb like) morphemes, naming 
(noun) morphemes, or connectors. Other possible 
grammar constructs used in sentences may include 
those based on "home" systems. Home systems are 
general-purpose gestural languages, whose grammar 
and syntax are not borrowed in any way from a host lan- 
guage. Examples of these languages are gestural lan- 
guages developed by deaf children of hearing parents 
who have not been exposed to American Sign Lan- 
guage (ASL), and the "plains talk" of North American 
Indians, which was used as a trade language. 
[0014] In one example, we provide a method for in- 



putting information to a computer connected to a de- 
formable piece that can be manipulated, and optionally 
to various position sensors (both relative and absolute), 
pressure sensors, thermal sensors, or even light sen- 
5 sors. The method comprises the steps of manipulating 
the deformable piece to provide a first morpheme input 
to the computer, with the first morpheme input normally 
triggering a first default action by the computer. The de- 
formable piece may also be manipulated to provide a 
10 second morpheme input to the computer, with the sec- 
ond morpheme input converting the normally triggered 
first default action to a second action. The first and sec- 
ond morphemes (and any subsequent morphemes) to- 
gether form a sentence that can be interpreted as a corn- 
's mand to implement a computer controlled action, wheth- 
er it be to unlock an electronically controlled door, dis- 
play a graphical image on a computer display, or begin 
logging on to a computer network. Advantageously, 
such a user interface system is well suited for interaction 
20 with small computer devices, and may even be cross- 
cultural to a limited extent, with ability to squeeze or 
pinch being universal human attributes. 
[0015] The present invention is particularly useful for 
portable computers that can be held in the palm of a 
2S hand. If the portable computer is partially or completely 
surrounded with a deformable material having embed- 
ded or contact pressure/deformation sensors, a user is 
able to gesture with or manipulate the whole computer 
in order to achieve some desired result. Material defor- 
ce? mation can be implemented at various scales. For in- 
stance, a computer and connected deformation sensors 
mounted inside a flexible tube could respond to right an- 
gle bending of the tube, or even complex knotting or 
looping of the tube. In practice however, only minute sur- 
35 face deformations are required, being just sufficient to 
provide tactile feedback to pressing, pinching, or bend- 
ing type manipulations. In either case, however, the 
measurement of location and pressure applied to a sur- 
face is sufficient to characterize the mode of interaction 
^0 (distinguishing a pinch from a prod). 

[0016] One particularly preferred embodiment of a 
handheld portable computer that responds to a physical 
manipulatory grammar in accordance with the present 
invention includes a computer, a feedback module to 
45 provide visual, auditory, or tactile feedback to a user (e. 
g„ processor connected LCD display, audio speaker, or 
tactile display to present Braille or other conventional 
touch interface), and co-mounted graspable, deforma- 
ble piece partially or completely surrounding the feed- 
so back module. In addition, various thermal or pressure 
sensors are mounted to detect handedness of a user, 
with the grasping hand generally being the non-domi- 
nant hand for the user. Depending on the handedness 
ol the user (which can be considered as a morpheme), 
55 the displayed data structure is modified. For example, 
text displayed on an LCD display screen may be auto- 
matically shifted rightward on the screen to allow pen 
based annotation on the left side of the screen, thereby 
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aiding left handed users. 

[0017] Physically manipulatable user interfaces addi- 
tionally provide an opportunity for multiple devices to in- 
teract in a user friendly manner For example, a tileable 
display system having multiple displays with touch sen- 
sitive deformable pieces surrounding the display can be 
used to organize documents based on relative position 
of contact of each display. For example, if two displays 
initially showing separate data structures (e.g. two dif- 
ferent pages from two different electronic books) are 
brought together in side by side touching contact, the 
displayed visual information can alter (e.g. by displaying 
adjacent pages of a single electronic book). As will be 
appreciated, one can also use multiple computers con- 
nected to physically manipulatable control elements to 
construct complex commands for organizing data struc- 
tures. 

[0018] Additional functions, objects, advantages, and 
features of the present invention will become apparent 
from consideration of the following description and 
drawings of preferred embodiments. 
[0019] Some examples of computers and interfaces 
according to the present invention will now be described 
with reference to the accompanying drawings, in which: - 

Figure 1 is a schematic diagram of zoomorphic 
computer in a shape of a teddy bear, the bear sup- 
porting visual and non-visual status displays, and a 
pressure sensor array for detecting surface defor- 
mations; 

Figure 2 is a graphical diagram illustrating various 
physical manipulation morphemes, with axes re- 
spectively illustrating complexity of senseme tuples 
required to form a morpheme, and plasticity of a de- 
vice required to support the physical manipulation; 
Figures 3-16 schematically illustrate various pre- 
ferred classes of physical manipulation mor- 
phemes; 

Figure 17 is a graphical diagram illustrating various 
spatial morphemes, with a first axis illustrating com- 
plexity of senseme tuples required to form a mor- 
pheme, and a second axis illustrating the degree of 
positional information needed to support the phys- 
ical manipulation (moving along a continuum from 
relative local measurement along a single specified 
dimension to absolute global measurement with six 
degrees of freedom determined); 
Figures 18-26 schematically illustrate various pre- 
ferred spatial manipulation morphemes; 
Figure 27 is a graphical illustration showing increas- 
ing degrees of sensor system complexity that can 
be used to detect various categories of environmen- 
tal stimuli, including light effects, thermal effects, 
the electromagnetic environment, and the vibratory/ 
acoustic environment; 

Figure 28 is a graphical diagram illustrating various 
physical manipulation morphemes for multiple in- 
teracting devices, with axes respectively illustrating 
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the complexity of senseme tuples required to form 
a morpheme, and increasing level of physical con- 
tact; 

Figures 29-34 schematically illustrate various pre- 
5 ferred manipulation morphemes for multiple inter- 
acting devices; 

Figures 35 and 36 are schematic diagrams illustrat- 
ing "squeeze" and lilt" morphemes applicable to a 
portable computer; 

Figure 37 is a schematic diagram illustrating tilt and 
squeeze morphemes used to control view of large 
two dimensional data sets with a relatively small dis- 
play ol a portable computer; 
Figure 33 is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a right handed user; 
Figure 39 is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a left handed user; 
Figure 40 is an electronic schematic illustrating 
components of the pressure and tilt sensitive mod- 
ules of a portable computer such as illustrated in 
Figures 35-39; 

Figures 41 and 42 are schematic illustrations of a 
scanner/printer/copier using a paper shaped dis- 
play interface to support morphemic input; 
Figure 43 is a schematic view of a tangible compu- 
ter interface formed as a stylized key, with rotation 
of the key enabling logon to a password protected 
personal computer; 

Figure 44 is a schematic view of a zoomorphic de- 
vice enabling logon to a password protected per- 
sonal computer; 

Figure 45 is a schematic illustration of a zoomorphic 
device in the shape of a penguin used as an audi- 
ovisual control system; and 
Figure 46 is a schematic illustration of the zoomorph 
of Figure 45 alternatively used in an electronic mail 
notification system. 

[0020] Figure 1 illustrates a teddy bear shaped 
zoomorphic embodiment of the present invention suita- 
ble for supporting a morphemic user interlace grammar. 
Support of the grammar can require detection of a user's 
physical manipulation of a zoomorphic device, detection 
of relative or absolute spatial location of the device, de- 
tection of various environmental factors acting on the 
device, and even detection and interaction with multiple 
devices or external computer networks. As illustrated, a 
zoomorphic device 1 0 has a deformable surface 20 with 
an underlying deformation sensor mesh 22 for detecting 
surface deformation across or within multiple subre- 
gions of the deformable surface 20. The deformation 
sensor mesh 22 is connected to an internally contained 
processor 24 having associated memory system 26. For 
detecting various positional or environmental variables, 
a sensing system 28 is also provided. The illustrated de- 
vice further includes a feedback module 33, which may 
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include an externally visible status display 30 or a non- 
visual feedback module 31 (typically delivering auditory 
or tactile feedback). In the illustrated device, a commu- 
nications system 32 for reception or transmission of in- 
formation to other electronic or computing devices is al- 
so provided. All these components can be powered by 
a power supply 25, which is usually an internally mount- 
ed rechargeable battery of conventional construction. 
[0021 ] Whatever the shape of device 1 0, for operation 
of the present invention the device 10 is completely or 
partially enveloped by the deformable shell or surface 
20. The present invention supports use of a great variety 
of designs and materials for the deformable surface 20, 
depending on the required plasticity, durability, longev- 
ity, and of course, cost constraints. For example, con- 
templated designs for deformable surface 20 include, 
but are not limited to: 
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bedded within, or positioned to contact, the deformable 
surface 20. The deformation sensor mesh 22 can in- 
clude an array of individual compressional or tensional 
strain sensors, or alternatively, embedded or attached 
5 positional sensors. For certain applications, continuous 
sensors (e.g. bilayer sheets of capacitance sensors) 
may be employed. One particularly useful continuous 
sensor type uses multiple capacitance or resistance 
strips, with deformation pressure resulting in a position- 
10 ally localizable analog signal proportional to the applied 
deformation pressure. Various sensor types can be 
used, including simple capacitance sensors, resistive 
strain sensors, analog or digital pressure switches, in- 
ductive sensors, or even fluid flow sensors. Depending 
is on the sensor type employed, sensor data can be direct- 
ly fed to the processor 24 in digital form, or be trans- 
formed to digital format by an general purpose analog/ 
digital converter that typically provides a 4 or 8 bit range 
(although as few as one or as many as 32 bits may be 
required by various applications). An analog to digital 
converter may be internal to the processor 24 or provid- 
ed as an external module. As will be appreciated, the 
sensor mesh 22 is intended to include combinations of 
sensors and sensor types, which can be used over the 
whole or part of the deformable surface 20. 
[0023] A positional or environmental sensor system 
28 can also be supported by zoomorphic device 10. Var- 
ious sensor modes can be supported, including abso- 
lute or relative positional information as determined by 
gyroscopic sensors, accelerometers, or acoustic or in- 
frared ranging techniques. Environmental sensors, in- 
cluding conventional light, image, thermal, electromag- 
netic, vibratory, or acoustic sensors can also be present. 
Depending on the desired application, even costly en- 
vironmental or positional sensors such as those incor- 
porating differential GPS positioning, image analysis or 
recognition, acoustic or voice identification, or differen- 
tial thermal sensors can be used as morphemic input. 
Such morphemic input, taken in conjunction with mor- 
phemic input as detected by sensor mesh 22, can en- 
hance precision and flexibility of a user's control of 
zoomorphic device 10. 

[0024] As illustrated, both sensor system 28 and sen- 
sor mesh 22 are connected to the processor 24 and as- 
sociated memory 26. The processor 24 and memory 26 
are typically mounted within the deformable surface 20, 
by either direct attachment to the deformable surface 20 
or by attachment to a hard casing positioned within the 
deformable surface 20. Conventional CISC or RISC 
processors can be used in the illustrated embodiment, 
with low power processors such as the Signetics 87c752 
or 87c751 , Motorola 68HC1 1 or 68582, or ARM 71 0 be- 
ing preferred, if convenient, coprocessors such as ana- 
log to digital converters or digital signal processors can 
be used alone or in conjunction with a main processor. 
Conventional flash, static, or dynamic RAM can used in 
the present invention, although for certain applications 
higher cost embedded DRAM may also be used. In 



plush cloth or textile material, optionally supported 
by foam or plastics, including closed or open celled 2° 
polymeric foam material having a wall thickness of 
millimeters to centimeters, with thinner walled em- 
bodiments being supported (e.g. by adhesive at- 
tachment) by an internal hard shell (constructed 
from polymeric or metallic materials), and those 25 
thicker walled embodiments directly supporting (by, 
e.g. brackets or supports) internal components 
such as processor 24. Suitable foams may include 
those composed in whole or in part of widely avail- 
able synthetic rubbers such as polychloroprene 30 
(neoprene), polystyrenes, rubber or nitrile rubber la- 
tex foams, polysiloxanes, block polymers including 
styrenebutadiene or styrene isoprene, or any other 
conventional material having good elasticity and 
deformability; 3s 
a thin single layer polymeric surface loosely 
wrapped around a internal hard shell (the hard shell 
being constructed from polymeric or metallic mate- 
rials). For example, a nylon or cotton weave, single 
layer polyethylene, synthetic rubber (with little or no 40 
foam cells present), or natural polymeric materials 
such as leather wrapped around a polystyrene cas- 
ing can be used; 

a composite layered surface having a durable pol- 
ymeric outer layer supported by an inner foam layer; 45 
or even 

a polymeric bilayer having an intermediate fluid or 
gel layer of a viscous or thixotropic material that can 
be used to support extreme deformations. The in- 
termediate layer can be relatively thick (one the or- so 
der of centimeters), or in certain embodiments can 
have a thickness measured on micron to millimeter 
scales. Such extremely thin layers would allow 
complex twisting, folding, curling, or crumpling ac- 
tions, and have been described in conjunction with ss 
U.S. Patent 5,389,945. 

[0022] The deformation sensor mesh 22 can be em- 
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some storage intensive applications, memory 26 can in- 
clude additional harddisk storage, either located within 
the device 10 or available through an external connec- 
tion. As will be appreciated, for many applications use 
of optional external communications can at least partial- 
ly supplant use of internal processors and memory (ex- 
cept for that necessary to support required sensor or 
communication buffering and signalling). 
[0025] The present invention optionally supports 
communications with an external computer system 40 
using its internal communications system 32 and asso- 
ciated transceiver 34. The external computer system 40 
also includes a transceiver 42, a personal computer or 
workstation 44, and is connected to a local or wide area 
network computer system 46. The transceivers 34 and 
42 can support various communication protocols and 
designs, including use of a serial tethered line 36 (using, 
for example the RS-232C interface protocols), use of in- 
frared signals 38 adhering to widely utilized IRDA com- 
munication standards, or use of radiofrequency signal 
37 (which can be, for example, a cellular telephone, 
900MHz radio, or digital PCS telephonic 
communications) . Alternative communication stand- 
ards, or even alternative communication carriers such 
as those based on optical or acoustic techniques, can 
of course be employed. 

[0026] As will be appreciated, in addition to direct 
communication with external computer system 40, the 
device 10 can be directly or indirectly maintained in con- 
tinuous or intermittent communication with a number of 
suitably equipped electronic devices, including a tablet 
computer 110, or even another zoomorphic portable 
computer 11 similar function to device 10. Communica- 
tion can be direct to a target device, or through an inter- 
mediary retransmitter such as computer system 40. 
Other possible communication targets include automa- 
tion control systems, security authorization units, per- 
sonal digital assistants, notebook computers, or any 
other suitably equipped electronic system. 
[0027] Results of communications with external de- 
vices, presentation of device stored information, or de- 
vice status updates can all be provided to a user through 
processor 24 controlled update of feedback module 33. 
Feedback to a user can be primarily visual, such as can 
occur in conjunction with visual display 30. Generally, 
the display 30 can be a conventional passive or active 
matrix liquid crystal display, although use of more so- 
phisticated (and expensive) displays based on various 
electrooptical or micromechanical techniques can of 
course be used. In addition, for certain devices a non- 
imaging display such as may be formed by a small 
number of status lights (e.g. red or green LEDs), or lo- 
calized or distributed chromatic changes (in conjunction 
with a deformable surface 22 constructed with suitable 
electrochromic materials) may be all that is necessary 
for visual feedback to the user. 
[0028] In some embodiments of the invention, visual 
output through display 30 may be augmented (or even 
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replaced) with a non-visual display 31 . The non-visual 
display 31 can include tactile displays based on internal 
actuators, auditory feedback, or even displays based on 
conformal changes in device appearance. For example, 

s one possible feedback display is based on internal au- 
ditory speakers (emitting a range of sounds from simple 
"beeps" to well formed speech, depending on available 
processor speed and functionality) for providing user 
feedback. As will be appreciated, non-visual display 31 

10 and its associated actuators or electronics can support 
alternative feedback modes, including, for example, 
force feedback to a user through internal actuators, tac- 
tile based feedback (e.g. with multiple surface projec- 
tions for presentation of Braille or other conventional 

is tactile user interface), modifications to the surface tex- 
ture of the device, or any other conventional mechanism 
for supplying information to a user. 
[0029] To better appreciate operation of the present 
invention, some selected modes of physical manipula- 
te? tion of device 10 are schematically presented in Figure 
1 . As seen in Figure 1, device 10 can be translationally 
moved in three spatial dimensions, as illustrated with 
reference to orthogonal force arrows 50, 51 , and 52. In 
addition to translational motion, the device 10 can be 

25 rotationally moved in any or all three spatial dimensions, 
as represented by arrows 53, 54, and 55. By use of sen- 
sor system 28 (alone or in combination with communi- 
cation system 32), the relative or absolute position and 
orientation in three dimensions can be determined. 

30 [0030] In addition to determining spatial position and 
orientation through use of sensors 28, the device 1 0 can 
optionally use sensor mesh 22 to measure and localize 
transient or continuing force application, with force vec- 
tors and related timing information being determined 

35 and interpreted. A few possible force actions (deforma- 
tion modes) are schematically illustrated in Figure 1, 
with arrows 60 and 61 illustrating depression of surface 
20 (with the combination representing a squeeze), ar- 
rows 62 and 63 illustrating sliding or rubbing deforma- 

40 tion (with the combination representing a twist), and the 
combination of sliding arrows 65 and 66, and outward 
pull 67, together representing a pinch and outward pull. 
The strength of the applied force can be measured (e. 
g. a hard or soft squeeze is differentiated), its spatial 

45 expanse found (e.g. to differentiate between poking with 
a fingertip or a thumb pad), and timing determined (e.g. 
to differentiate between a quick or a slow depression of 
the surface). The deformation so caused may be either 
permanent or transitory. 

so [0031] As will be appreciated by those skilled in the 
art, each of the foregoing force actions represented by 
arrows can be considered a senseme. Some temporally 
distinguishable sensemes (or combinations of sense- 
mes such as the foregoing discussed pinch/pull combi- 

55 nation) further represent morphemes used as a basis 
for a morphemic grammar in accordance with the 
present invention. All of the following described mor- 
phemes can be modified by numerous variations in ap- 
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plied pressure, force utilized, appendages used, body 
parts, or extraneous intermediary objects used to apply 
force. In addition, timing of various objects (whether 
quick, slow, or alternately quick and slow) can modify 
interpretation o1 a morpheme. For example, if "squeeze" 5 
is taken as atypical morpheme, one can appreciate var- 
ious squeeze operations such as quick squeeze, slow 
squeeze, hard squeeze, soft squeeze, narrow squeeze, 
wide squeeze, squeeze between hands, squeeze be- 
tween one hand and a user's chest or head, squeeze 10 
between one hand and a table or wall, a squeeze made 
between two pens or two books, or even a squeeze be- 
tween a user's tongue and the roof of the mouth. For 
purposes of the present invention, all such squeeze 
morphemes would be considered members of the 15 
"squeeze" class, with individual variations acting as pos- 
sible modifiers or selected cases, just as a "house cat" 
might be considered a particular member of the class of 
"felines", which more generally embraces lions, tigers, 
and bobcats. 20 
[0032] To aid in understanding the diversity of con- 
templated physical manipulation morphemes, Figure 2 
illustrates selected morphemes arranged by increasing 
plasticity of the device required to enable morpheme uti- 
lization, and by increasing complexity of available ?s 
senseme tuples required to form or interpret a mor- 
pheme applied toa particular class ol device. Beginning 
with the least plastic device and the simplest senseme 
set used to compose a morpheme, a definition of a pos- 
sible physical manipulation and typical function invoked 30 
by that manipulation of a device similar (but of course 
possibly more complex) to that described in conjunction 
with Figure 1 is presented: 

DEPRESS • 35 

[0033] Definition: Indenting one or more subregions 
of the device through the application of pressure. 
[0034] Example: As seen in Figure 3, consider a de- 
vice 1 22 having a display 1 23. The device 1 22 supports 40 
a graphical illustration software application which allows 
users to position geometric shapes or user defined 
graphical objects. The device can have four pads 124 
placed around it, one on each side. By deforming a par- 
ticular side, the user indicates a desire to "nudge" the 45 
currently selected geometric shape 1 25 away from that 
side to a new position 126. 

SQUEEZE 

so 

[0035] Definition: To deform one or more subregions 
of a device through the application of vectors of force, 
some components of which are directed towards each 
other, which compress the morphological structure of 
the device. This is a special case of depression. 55 
[0036] Example: As illustrated in Figure 4, consider a 
device 132 which can display one or more documents 
in either iconic (closed) or textual display (open) modes. 



By selecting an open document 135, and then squeez- 
ing a deformable edge 134 of the device 132, the user 
indicates a desire to make the document "smaller", in 
this case to iconize it as icon 1 36. 

FOLD 

[0037] Definition: To deform a second subregion by 
bending the first subregion such that it partially or com- 
pletely overlaps the second subregion. Further defor- 
mations can be applied to that new morphological struc- 
ture on other subregions. 

[003S] Example: As illustrated in Figure 5, consider a 
device 142 which can display documents. Suppose this 
device 1 42 has been augmented such that a deformable 
horizontal "flap" 144 on atop edge of the device 142 can 
be folded to partially obscure a display 143. When the 
user makes this folding gesture, the user indicates a de- 
sire to password-protect ("hide") the currently displayed 
document. 

CURL 

[0039] Definition: Deforming one or more subregions 
of the device by spirally configuring subregions relative 
to each other in a cylindrical or circular morphological 
structure. 

[0040] Example: As illustrated in Figure 6, consider a 
device 1 50 which can display documents in various lan- 
guages (English, French, etc.). When the user takes 
such a device 1 50, curls in a direction indicated by arrow 
1 57 into a tube, and then uncurls it, this "abra-cadabra" 
gesture tells the device 150 to display the current doc- 
ument in a different language. 

STRETCH 

[0041] Definition: Deforming one or more subregions 
of a device through the application of vectors of forces, 
some components of which are directed away from each 
other, the vectors being applied at opposite ends of the 
device. 

[0042] Example: As illustrated in Figure 7, consider a 
device 160 with a graphical software application which 
allows users to manipulate geometric shapes. By 
stretching the device 160, the user indicates a desire to 
"resize" or "rescale" the currently displayed shapel65 to 
a larger size 1 66, the amount of resizing being a function 
of the amount of deformation. Note that SQUEEZING 
can indicate resizing the currently displayed to a smaller 
size. 

PINCH 

[0043] Definition: Manipulating one or more subre- 
gions by applying vectors of forces, aligned directly to- 
wards each other, on opposite sides of the affected su- 
bregion(s). This is typically, but not exclusively, accom- 
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plished using two-finger tactile force. PINCH is a special 
case of SQUEEZE. 

[0044] Example: As illustrated in Figure 8, consider a 
device 170 which can copy documents. By performing 
a "pinching 1 action 1 75, the user indicates that they wish 
the next set of copies to be issued in stapled form. 

DOGEAR 

[0045] Definition: Deforming a second subregion by 
folding a first subregion at a logical corner or edge of 
the second subregion, indicating a marker or location 
point (e.g. a bookmark) to be later referenced. DOGEAR 
is a special case of FOLD. 

[0046] Example: As illustrated in Figure 9, consider a 
device 1 80 which displays a subset of the pages from a 
multi-page document. By "dogearing" the upper right 
corner 185 of device 180, the user indicates that they 
wish a bookmark associated with the currently displayed 
page or pages. 

TWIST 

[0047] Definition: Deforming one or more subregions 
of the device through the application of two opposing 
rotational forces offset from each other by a non-zero 
difference about some central axis. 
[0048] Example: As illustrated in Figure 10, consider 
a device 190 which, over time, becomes degraded in 
some aspect of its performance (its disk becomes frag- 
mented, its memory needs garbage-collection, etc.). By 
performing a "TWIST" gesture 195, the user indicates 
that they wish the device to "wring itself out", performing, 
for example, garbage collection. 

RELIEF-MAP 

[0049] Definition: Deforming one or more subregions 
of the device by raising and or lowering them by either 
spatial transformation or the addition/removal of mate- 
rial. 

[0050] Example: As illustrated in Figure 11, consider 
a device 200 which can display documents in either one- 
ortwo- page format. When the user "scores 0 the device 
by making a vertical indentation about the center axis of 
the device while it displays a single page 206, the device 
200 interprets the morpheme to request display of doc- 
uments in a two-page format as pages 207 and 208. 

RIP 

[0051] Definition: Deforming one or more subregions 
of the device by introducing a spatial discontinuity, by 
applying vectors of force to partially or totally disconnect 
these subregions from the device. 
[0052] Example: As illustrated in Figure 12, consider 
a device 210 which can copy some or all of its informa- 
tion. When the user performs the "ripping" gesture 215, 



removing one or more subregions, the device 210 cop- 
ies its currently selected data set onto those subregions 
216 and 217. 

5 PERFORATE 

[0053] Definition: Deforming one or more subregions 
of the device by means of introducing a change in the 
spatial connectivity of the subregions such that a hole 
10 is introduced (either temporarily or permanently) in the 
device. 

[0054] Example: As illustrated in Figure 13, consider 
a device 220 which is used to route messages between 
various parts of a device network, and which displays 
is this functionality to the user by means of lines 222 rep- 
resenting message pathways. When the user perforates 
the device 220 with a finger or object 224 to introduce 
a hole in one of these pathways 222, the system stops 
routing messages along that pathway. 

20 

SIMILARITY 

[0055] Definition: Spatial motion, and/or physical de- 
formation of one or more subregions, of a device previ- 

25 ously configured to represent some other predefined ob- 
ject or system. Typically the device acts in a manner 
consistent with the behavior of the real-world object 
when it is manipulated in this manner Such avatars, tal- 
ismans, or other computationally enabled devices 

30 present themselves to a user as a physical instantiation 
of another object or system. Avatars can be zoomorphic 
(including animal or human shapes) or configured to 
mimic real world objects with any desired degree of spe- 
cificity. 

35 [0056] Example: As illustrated in Figure 14, consider 
a device 230 capable of text-to-speech audio output, au- 
dio input, and visual media presentation. The device 230 
presents itself to a user in an anatomically accurate 
shape of a human head. When the user opens lips 232 

40 on the human head, internal sensors detect opening of 
the lips and activate text-to-speech capability. When the 
user covers or closes the eyes 234, associated cameras 
or video systems are disabled, while covering ears 236 
results in disabling audio recording systems. 

45 

3D MAP 

[0057] Definition: The morphing of a device that can 
be molded around an external object to permit a sensor 

50 mesh to determine simultaneously size and shape of the 
external object. The range of possible external objects 
is large but is limited by the size of the device's solid 
inner-housing and the volume of the moldable material 
in the outer housing. In this system the device has the 

55 ability to accurately sense the amount of material from 
its inner surface to the outer edge of the moldable ma- 
terial (example: via ultrasonic sounding, similar to sonar 
in water) thus determining an accurate electronic model 
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for the shape of the molded enclosure. 
[0058] Example: As illustrated in Figure 15, by press- 
ing a device attached moldable material 244 around the 
surface of an external object (e.g. a cog 242) a device 
240 can automatically generate a CAD model of that ob- 
ject and store it in its memory. 

MIMICRY 

[0059] Definition: Deforming one or more subregions 
of the device such that the resultant morphological 
structure resembles a known real-world object and 
through this association of subregions, the device acts 
in a manner consistent with the object it resembles. 
[0060] Example: As illustrated in Figure 16, consider 
a device 250 containing a computer which contains text- 
to-speech and audio input capability, and which 
presents itself to the user as a misshapen blob having 
the consistency and plasticity of moldable putty or clay. 
When the user performs the U MIMICRY M action by mold- 
ing part of the device 250 to resemble an ear, the audio 
input capability is activated. 

[0061] In addition to morphemes based on physical 
manipulation, various morphemes based on varying de- 
grees of relative or absolute spatial positioning are con- 
templated to be uselul in practice of the present inven- 
tion. To aid in understanding the diversity of contemplat- 
ed spatial morphemes, Figure 17 illustrates selected 
spatial morphemes arranged by increasing knowledge 
of spatial position required to enable morpheme utiliza- 
tion, and by increasing complexity of available senseme 
tuples required to form or interpret a morpheme applied 
to a particular class of device. Beginning with a device 
having only rudimentary relative positioning functional- 
ity for supporting simple spatial sensemes, and ending 
with a device absolutely positionable to within centime- 
ters anywhere on Earth, a definition of a possible spatial 
manipulation and typical function invoked by that ma- 
nipulation of a device similar (but of course possibly 
more complex) to that described in conjunction with Fig- 
ure 1 is presented: 

TRANSLATE (relative to device) 

[0062] Definition: The linear movement of a device's 
center of mass from one position in space to another. 
[0063] Example: Used to substitute for mouse control- 
led graphical "sliders" in conventional graphical user in- 
terfaces. When it is only physically possible to display a 
small amount of list, large listings can still be searched 
by "scrolling" a display window in response to the 
TRANSLATE morpheme. 

SHAKE 

[0064] Definition: Spatially translating all subregions 
of a device by repeated movement in opposing direc- 
tions, such that the net translation is negligible. 
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[0065] Example: As illustrated in Figure 13, consider 
a device 260 which is used as a calculating device. 
When the user performs the "SHAKE" gesture, the de- 
vice 260 clears its accumulator. 

5 

REVOLVE 

[0066] Definition: Rotating all subregions of a device 
by rotating the subregions about a point internal to the 
io device, about any arbitrary plane. 

[0067] Example: As illustrated in Figure 19, consider 
a device 270 which displays an imaged slice of volumet- 
ric data, such as medical data from a series of CAT 
scans. By rotating the device about a center point 272 
75 internal to the device 270 to a new position 274, the 
plan e specifying the imaged slice is changed according- 
ly. 

TILT 

[0068] Definition: Rotating one or more subregions of 
a device by rotating the subregions such that one or 
more components of the rotary force are in the direction 
of gravity and the amount of rotation is between about 
-180 degrees and +180 degrees. 
[0069] Example: Consider a device which displays 
frames from an animation sequence on the side facing 
the user. As the device is tilted away, the speed of the 
animation increases - as its tilted towards the user, the 
speed of the animation decreases, analogous to oper- 
ation of a gas pedal. 

FLICK 

[0070] Definition: A forwards TILT immediately fol- 
lowed by an opposing backwards TILT. 
[0071] Example: As illustrated in Figure 20, consider 
a device 280 which can transmit some subset of its data 
to another device. When the user performs the "FLICK" 
gesture by quickly tilting in the direction ol arrow 282, 
followed by a reverse tilt along arrow 282, the device 
280 performs this transmission, towards the device (not 
shown) pointed to by the ray of the gesture. 



[0072] Definition: Rotating one or more subregions of 
the device by rotating the subregions about a point in- 
ternal to the device, such that the plane of rotation is 
one of the device's surface planes. SPIN can be consid- 
ered a special case of REVOLVE. 
[0073] Example: As illustrated in Figure 21, consider 
a device 290 which can display a frame of video 295 
from a video sequence. The user performs the "SPIN" 
gesture in a counter-clockwise direction, the device dis- 
plays earlier frames in the sequence; when the gesture 
is performed in a clockwise direction (arrow 292), the 
device 290 displays a later frame 296 in the sequence 
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(represented by film strip 294). 
ORIENT 

[0074] Definition: Rotating one or more subregions of 
the device by rotating the subregions about the center 
of the device, such that the plane of rotation is one of 
the device's surface planes, and the amount of the ro- 
tation is a multiple of 90 degrees (i.e. to rotate the device 
between cardinal compass points). Orient can be con- 
sidered a special case of SPIN, which in turn is a special 
case of REVOLVE. 

[0075] Example: As illustrated in Figure 22, consider 
a device 300 which can display a document in either 1 
page, 2-page, or 4-page format ("1-up", °2-up° t or 
"4-up") . When the user performs the orient gesture in a 
clockwise direction (arrow 302), the device 300 increas- 
es the number of document pages it is displaying from 
one page 305 to two pages 306 and 307. Further orient 
gestures would increase the number of displayed pag- 
es. When performed in a counter-clockwise direction, 
the device 300 decreases the number of pages it is dis- 
playing. 

FACE 

[0076] Definition: Manipulating one or more subre- 
gions of the device such that a first set of subregions is 
no longer bottom-most and a second distinct set of su- 
bregions now assumes the first subregions former po- 
sition. 

[0077] Example: As illustrated in Figure 23, consider 
a device 310 which displays documents, and which al- 
lows users to edit such documents. Consider further the 
case in which the device presents itself to the user in 
the form of a cube, in which 6 different documents are 
displayed on the 6 different faces. When the user per- 
forms the "FACE" gesture by making a particular face 
top-most, the document which is now on the top-most 
face becomes editable by the user, while the document 
which is no longer top-most is no longer editable. 

LIFT 

[0078] Definition: The movement of a device's center 
of mass in a direction opposite to the current gravitation 
force acting on the device. 

[0079] Example: Commanding the device to display 
the computer's file system at a position one level higher 
in the hierarchy. 

PAN 

[0080] Definition: The application of a TRANSLATION 
to a device such that it is moved parallel to the front of 
the user's body, at a substantially constant height. 
[0081] Example: Viewing a spread-sheet on a device 
with a display so small only one cell can be shown. By 



PANing the device, the contents of the current row can 
be shown in sequence depending on the rate or amount 
ol PAN. However, if the device were rotated away from 
its current orientation during the PAN, a new row would 
5 be chosen. The selection of the row could be dependent 
on the deviation from the original orientation. 

PUSH-PULL 

10 [0082] Definition: Manipulating one or more subre- 
gions of the device by spatially translating them such 
that they are moved along a line of projection from the 
center of the device to the vertical axis of the user's 
body. 

75 [0083] Example: Consider a device with audio output 
capability. As the device is "pushed" further away from 
the body, its audio output level increases. When it's 
"pulled" towards the body, its level decreases. 

WHACK 

[0084] Definition: The application of an accelerative 
or de-accelerative force to one or more subregions of a 
device such that said subregions contact or are contact- 
ed by an external object, causing an equal and opposite 
countering force. 

[0085] Example: As illustrated in Figure 24, consider 
a device 320 which can perform long and unpredictable 
database searches. When the user performs the 
WHACK gesture (e.g. upon a table 322), the current 
search is aborted. 

ORIENT (relative to environment) 

[0086] Definition: Manipulating two subregions of the 
device such that the line drawn between the centers of 
those two subregions alters its orientation with respect 
to the surrounding environment. 
[0087] Example: Displaying a CAD drawing of a ma- 
chine-part in 3D on the display ol a mobile device. As 
the orientation of the device changes, so too does the 
viewing angle and position of the rendered image. 

ORBIT 

[0088] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, about any arbitrary axis. 
[0089] Example: As illustrated in Figure 25, consider 
a device 330 which can search a network database, 
such as the World Wide Web, for information. When the 
user performs the "orbit" gesture, such a search is initi- 
ated. The radius 332 of the rotation 334 specifies the 
breadth of the search - wider circles specify a wider 
search. The speed of the gesture specifies the time limit 
imposed on the search - the quicker the gesture, the 
more cursory the search. 
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ORBIT RELATIVE TO USER 

[0090] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, where said point is proximal to a 
body feature of the user. This is a special case of ORBIT 
[0091] Example: As illustrated in Figure 26, consider 
a device 340 which can perform audio output. By per- 
forming the ORBIT gesture (in direction indicated by ar- 
row 344) about the user's ear 345, the audio output is 
activated. 

MOVEMENT IN ROOM 

[0092] Definition: The local detection of a device's 3D 
position relative to reference points found within an en- 
closing room. Differences in the measured position are 
used to trigger actions. 

[0093] Example: A virtual filing system that allows you 
to save and restore files based on the devices current 
position in the room. To save a file you might think care- 
fully about the contents of the file and then walk to the 
position in the room that might be most easily associated 
with it. When retrieving the file you would use the same 
thought processes and go back to the position you had 
associated with the file. On doing so the files associated 
with that position would be displayed and you would 
then be able to select the file you were looking for. The 
system is useful because the human mind is very good 
at remembering information that is spatially organized 
rather than in some abstract information data structure. 

MOVEMENT BETWEEN WIDELY SEPARATED SITES 

[0094] Definition: Manipulating one or subregions of 
the device such that the detected absolute spatial posi- 
tion of those subregions is changed. 
[0095] Example: Consider a device which can display 
information from a database of client information. When 
the device is moved to a different client site, the device 
automatically updates its display to display information 
for the nearest client site. 

[0096] In addition to morphemes based on physical 
manipulation or spatial positioning, various morphemes 
based on sensed environmental conditions are contem- 
plated to be useful in practice of the present invention. 
To aid in understanding the diversity of contemplated 
environmental morphemes, Figure 27 illustrates select- 
ed environmental morphemes loosely arranged in order 
of increasing sensor complexity needed in some com- 
monly sensed environmental categories. For each of the 
proffered categories, some selected sensing systems 
supportable by a device such as that described in con- 
junction with Figure 1 are presented: 



20 
LIGHT 

[0097] Definition: Manipulating one or more subre- 
gions of the device such that the amount of light falling 

5 upon those subregions changes. 

[0098] Example: Consider a device used in a lecture 
hall to take notes. When the room lights are turned on, 
the light sensors detect this and turn down the backlight 
to conserve energy. When the room lights are turned off 

10 (e.g. during a slide show) the light sensors detect this 
and turn up the backlight to increase viewability. 
[0099] Light sensors can range from thresholded bi- 
nary light detectors, to light pattern detectors, to full im- 
aging systems. Advanced techniques can include im- 

15 age analysis and recognition to identify objects or per- 
sons. 

HEAT 

20 [0100] Definition: Manipulating one or more subre- 
gions of the device such that the amount of heat applied 
to those subregions changes. 
[01 01] Example: Consider a portable computer which 
has a stylus for entering text. By looking at the heat pro- 
25 file along the back surface of the computer, the compu- 
ter can detect whether it is being held with the left hand, 
the right hand, both hands, or neither hand, and update 
its interface accordingly. 

[01 02] Thermal (heat) sensors can range from simple 
30 temperature sensors to sophisticated differential ther- 
mal mappers and thermal imagers. 

ELECTROMAGNETIC 

35 [0103] Definition: Manipulating one or more subre- 
gions of the device such that the electromagnetic spec- 
trum applied to those subregions changes. 
[0104] Example: By analyzing the radio spectrum, the 
device can derive estimates as to its absolute spatial 

40 position, and use that to alter its functionality. 

[0105] Electromagnetic detection can include mag- 
netic compasses, radio detection, or GPS signal detec- 
tion. More advanced techniques can include electro- 
magnetic spectrum analysis and interpretation, such as 

45 roughly determining location based on available radio 
signals. 

VIBRATE 

so [0106] Definition: Manipulating one or more subre- 
gions of the device by vibration. 
[0107] Example: Consider a device which displays 
textual information. When the user takes the device on 
the bus, the ambient vibration level sensed by the device 
55 changes, and the device increases the size of the dis- 
played text to help the user compensate. 
[0108] This class of environmental morphemes can 
include detection of intermittent contacts, low frequency 
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rumblings, or acoustic level detection. More advanced 
techniques requiring greater processor power include 
maximum frequency identification, spectral analysis of 
acoustic frequencies (enabling the device to distinguish 
background environmental noises from speech, for ex- 
ample), or even speech based identification of persons 
in the vicinity of a device. 

[0109] In addition to morphemes based on physical 
manipulation, spatial position, or sensed environmental 
factors, various morphemes based on cooperation be- 
tween multiple interacting devices are contemplated to 
be useful in practice of the present invention. To aid in 
understanding the diversity of contemplated spatial 
morphemes, Figure 26 illustrates muitidevice mor- 
phemes arranged by an increasing level of possible 
physical contact, and by increasing complexity of avail- 
able senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 
with a device having only rudimentary edge deformation 
functionality for supporting simple spatial sensemes, 
and ending with complex deformable or embeddable 
devices that can be wrapped about each other, a defi- 
nition of a possible muitidevice manipulation and typical 
function invoked by that muitidevice manipulation of de- 
vices similar (but of course possibly more complex) to 
that described in conjunction with Figure 1 is presented: 

TOUCH 

[0110] Definition: To move one or more subregions of 
a device such that they enter physical contact with a su- 
bregion of a second device, in any alignment and to any 
extent. Or, to take two devices so aligned and remove 
that alignment. 

[0111] Example: Consider two portable computers 
350 and 351 , the first of which contains a database, and 
the second of which contains an IRDA port. When the 
user touches the first computer 350 to the second com- 
puter 351, the database is transmitted via the second 
computer's port. 

MATCH 

[0112] Definition: To move one or more subregions of 
a device such that they enter physical contact with one 
or more subregions of a second device, with the subre- 
gions) of the first device and the subregion(s) of the 
second device being aligned along one or more edges. 
Or, to take two devices so aligned and remove that align- 
ment. 

[0113] Example: As illustrated in Figure 30, consider, 
multiple devices 360, 361, and 362, which contain dif- 
ferent versions of the same basic database. When the 
user MATCHes the first device 360 to the second device 
361 , followed by matching the third device 362 to the 
matched first and second devices, their databases are 
reconciled (synchronized). 



STACK 

[0114] Definition: To move one or more subregions of 
a device such that they enter physical contact with a su- 
5 bregion of a second device, such that the first device is 
now located above, but physically adjacent to, the sec- 
ond device. Or, to take two devices so aligned and re- 
move (i.e. unstack) that alignment. 
[0115] Example: As illustrated in Figure 31, consider 
a set of devices 370, 371, and 372, each of which is 
displaying a frame of video from a longer video se- 
quence. When the devices are stacked, the ordering of 
the stacking specifies an order for the video editing, and 
a single composite video is now produced. 

TILE 

[0116] Definition: To move one or more subregions of 
a device such that they physically contact with a subre- 
gion of a second device, such that the first device and 
second device now form a single seamless spatial unit 
or to take two devices so aligned and remove that align- 
ment. TILE is a special case of MATCH. 
[0117] Example: As illustrated in Figure 32, consider 
a set of devices 380, 381, 382, 383, each of which can 
independently display a portion of a large photograph. 
When the devices are tiled, each device displays the 
portion of the photograph appropriate to its current rel- 
ative position in the tiled grid. 

RELATIVELY ALIGN 

[011 8] Definition: To move one or more subregions of 
a device such that they engage in a particular spatial 
relation to one or more other devices, where said devic- 
es are not touching. 

[0119] Example: As illustrated in Figure 33, consider 
a set of devices 390, 391 , 392, and 393 which are dis- 
playing a multi-page document. Whichever device is 
presently placed at the far left (device 390) displays the 
table of contents, whichever one is presently placed at 
the far right (device 393) displays the index, and the oth- 
ers display pages according to their respective loca- 
tions. As different devices can have different display ca- 
pabilities, moving them about can alter the document 
display. For example, if only one of the devices has a 
color display, when it is moved from second position to 
third position then (a) the device which was in third po- 
sition, displaying page #2, now displays page #1 , and 
(b) the color display, which was displaying page #1 , now 
displays page #2 in color. 

WRAPPING/EMBEDDING 

[0120] Definition: Manipulating one or more subre- 
gions of a device such that these subregions spatially 
occlude or are spatially occluded by some portion of the 
second device. 
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[0121] Example: As illustrated in Figure 34, consider 
a first device 400 which contains the infrastructure to 
support the filtering of email. Consider a second set of 
devices 401 and 402 which implement particular email 
filters. When the user physically embeds device 401 (or 
402) into the first device 400, by which the first device 
400 now wraps the second device 401 (or 402), the par- 
ticular email filter supported by the second device is ac- 
tivated. 

[0122] As those skilled in the art will appreciate, com- 
binations ol any of the foregoing described morphemes 
based on physical manipulation, spatial position, envi- 
ronmental conditions, or multiple interacting devices 
can be extended by participation in a morphemic "sen- 
tence". A sentence is defined as a sequence of one or 
more temporally disjoint morphemes. Typically, be- 
tween about 1/1 0th of a second and 2-3 seconds suffic- 
es to distinguish morphemes within a sentence. Of 
course, in some circumstances and indefinite time peri- 
od may elapse. The sentence level allows definition of 
a physical manipulatory grammar by appropriate choice 
of a morpheme sequence, and corollary rules govern- 
ing, for example, use of active (verb like) morphemes, 
naming (noun) morphemes, or connectors. Just as the 
position and relation of words in a sentence define the 
sentence's meaning (e.g. "horse chestnut" is not the 
same as "chestnut horse 11 ), similarly the position and re- 
lation of morphemes in a manipulatory sentence define 
the sentence's meaning. For example, in a communica- 
tion mode a FLICK followed by a WHACK could mean 
"transfer data and erase the local copy", while a WHACK 
followed by a FLICK means "power on the device and 
transfer data". In other settings, a FLICK or a WHACK 
could mean something entirely different. To better un- 
derstand construction of morphemic sentences, the fol- 
lowing examples are discussed: 

DATA TRANSFER SENTENCE 

[0123] Consider a device which can transmit some or 
all of its information to another device. Furthermore, this 
transmission can be done either unencrypted, or en- 
crypted (to increase security). Furthermore, this trans- 
mission of a text+graphics document can either include 
the graphics, or omit them (to save time). Suppose that 
the user wishes to perform the command "Transmit the 
information in document A, encrypted, omitting graph- 
ics, to machine B". Then a gestural sequence (morphe- 
mic sentence) to support this could be: 

DEPRESS - the user presses on a displayed rep- 
resentation of A, indicating that A is to be selected 
for an upcoming operation 

FLICK - the user flicks the device in the direction of 
device B, indicating that the operation is a transmis- 
sion to B 

FOLD - the user folds the top quarter of the device 
over the lower three-quarters, indicating that the 



transmission is to be encrypted. 
TWIST - the user twists the device about its central 
axis, indicating that the data is to be "wrung out", i. 
e. the graphics are to be omitted 
5 SQUEEZE - the user squeezes the device, indicat- 
ing that they are confirming that the operation is tc 
proceed. 

Note that none of these gestures, in isolation, per- 
forms an act - the "ensemble" of temporally sepa- 
io rated morphemes must be interpreted in order to 
form the complete action. 

DRAWING MODIFICATION SENTENCE 

is [0124] Consider a device which displays geometric 
shapes for user manipulation. Furthermore, one of the 
manipulations supported is to resize (or rescale) a 
shape. Furthermore, suppose that this resizing can be 
done either aliased (jaggy) or anti-aliased (edges are 

20 smoothed). Suppose that the user wishes to perform the 
command "Resize shape A by 120%, about the X axis 
only, using anti-aliasing". Then a morphemic sentence 
to support this could be: 

2S DEPRESS- the user presses on a displayed repre- 
sentation ol A, indicating that A is to be selected for 
an upcoming operation 

STRETCH - the user stretches some portion of the 
device, indicating that the operation is to be a re- 
30 size. When the user starts stretching, a portion of 
the status display displays "100". The user contin- 
ues to stretch until Ihe status display reads "120° 
RELIEF-MAP - the user "scores" the display by 
making a horizontal line of depression, indicating 
35 that the operation is to take place only about the 
horizontal (X) axis. 

DEPRESS - a circular thumb stroke is made in a 
different area of the device, indicating that anti- 
aliasing (smoothing the edges) is to be performed. 

40 

DATABASE PRESENTATION SENTENCE 

[01 25] Consider a device which contains various per- 
sonal information databases, such as a list of phone 
45 numbers, a list of addresses, and a calendar. Suppose 
that the user wishes the most appropriate of those da- 
tabases displayed. Then a gestural sequence to support 
this could be: 

50 SPATIAL LOCATION - the user carries the device 
such that it is spatially proximate to either the tele- 
phone, the address book, or the refrigerator (where 
the family calendar is displayed), whichever is ap- 
propriate. 

55 DEPRESS - the user touches the device to activate 
the operation. 

[01 26] The device now displays the personal informa- 
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tion appropriate to that location. 

DATABASE RETRIEVAL SENTENCE 

[01 27] To extend the foregoing example of a database 
presentation sentence, consider two computers which 
contain calendar databases. If the user wishes to syn- 
chronize the calendars, a suitable gestural sequence to 
support this could be: 

SQUEEZE - the user squeezes the device to acti- 
vates its gestural recognition capabilities 
ORBIT - the user orbits the device about the surface 
of the other device 3 times, indicating a desire to 
only match data for the next 3 weeks. 
MATCH - the user matches the edge of the device 
to the edge of the calendar, indicating a desire to 
"match" contents between the two devices. 

PRINTER/COPIER CONTROL SENTENCE 

[0128] Consider a device which can produce paper 
copies of documents. Suppose that the user wishes to 
tell such a device to produce a stapled, two sided copy 
of document A, enlarged to the next greater size. Then 
a gestural sequence to support this could be: 

DEPRESS - the user presses on a displayed rep- 
resentation of A, indicating that A is to be selected 
for an upcoming operation 
RIP - the user introduces a spatial discontinuity into 
a portion of the device, indicating that the upcoming 
operation is to be a copy ("carrying away" some of 
the data). 

PINCH - the user pinches the upper left corner of 
the device, indicating that the copies are to be sta- 
pled. 

SQUEEZE - the user presses on the front and back 
of the device, indicating that the copy is to be two- 
sided. 

STRETCH - the user stretches the device, indicat- 
ing that the copy is to be an enlargement to the next 
greater size. 

FACE - the device typically has its paper emitter on 
the bottom, preventing users from making copies 
accidentally. By facing the device such that the 
emitter is on the side, the copying operation is initi- 
ated. 

LIGHT BASED CONTROL SENTENCE 

[0129] Consider a device which can display docu- 
ments. Suppose the user is using the document while 
seated on a train, and wishes the document to display 
itself with a backlight when the train enters a tunnel, and 
wishes the document to display itself in a larger font 
when the train rumbles over rough tracks. Then a ges- 
tural sequence to support this could be 



SQUEEZE - the user squeezes on the device, indi- 
cating that a loss ol light is to be compensated for 
by a backlight. 

LIGHT - as the train enters a tunnel, the LIGHT ges- 
5 ture is made, and the device turns on the backlight. 
WHACK - the user briskly raps the device against 
the palm of their hand, indicating that their desired 
preference for adjusting hard-to-read documents is 
to increase the font size. 
10 VIBRATE - as the train goes over a bridge, the vi- 
bratory gesture is sensed. Because of the position 
of this VIBRATE morpheme (after the preceding 
WHACK gesture) in this morphemic sentence, the 
device now increases the font size on the displayed 
1$ text. 

LIGHT - the user puts the device into his suitcase, 
making the LIGHT gesture. In this context (with no 
SQUEEZE before it), the LIGHT gesture causes the 
device to power-off its display. 

[01 30] To better appreciate utility and construction of 
devices in accordance with the present invention, sev- 
eral examples of devices are now described: 

PORTABLE COMPUTER WITH SQUEEZE AND TILT 
CONTROL 

[0131] A handheld portable computer 500 (e.g. a 
3Com ® PalmPilot ® ) capable of being fitted with de- 
formable, pressure sensitive edging 504 is schematical- 
ly illustrated in Figures 35 and 36. The computer 500 
supports a name and address software application, pro- 
viding a user viewable name-address entry field on dis- 
play 503. In this embodiment, a user can squeeze the 
deformable, pressure sensitive edging 504 (squeeze ar- 
rows 507) of the computer 500. In response, the name 
and address software application causes the display 
503 to animate by slowly incrementing (scrolling) 
through the name list from "A" towards "Z" entries. When 
the user squeezes edging 504 again, the software ap- 
plication stops the scrolling animation. Scrolling func- 
tionality is further enhanced by the use of a tilt sensor, 
which allows the computer's behavior to mimic conven- 
tional rotatable address books. If the computer 500 is 
tilted away from the 45 degree angle at which someone 
might typically hold it, the scrolling rate is increased. In 
this application, the closer the computer 500 was tilted 
towards the user (as indicated by arrow 506 of Figure 
36), the faster the scroll rate toward n Z a . However, if a 
user tilted the computer 500 back past the neutral 45 
degree position (as indicated by arrow 506 in Figure 36), 
the animation would move backwards with a velocity re- 
lated to the magnitude of tilt. In this way it was possible 
for a user to search for items in a long list in a very natural 
way, while only using one hand. 
[0132] In an alternative mode schematically illustrat- 
ed in Figure 37, scrolling speed can be completely con- 
trolled by pressure. The greater the squeeze pressure 
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(arrows 537), the faster the list scrolls. Release of the 
applied pressure causes the scrolling to halt. In this al- 
ternative user interface strategy, application tilt (as indi- 
cated by orthogonal tilt arrows 530 and 532) could be 
used to change the direction of the scrolling through the 
list, allowing a user to search portions of a large two di- 
mensional data set (schematically illustrated as a data 
plane 520) that is not entirely visible either horizontally 
or vertically on display 503. By simply tilting the display 
503 of computer 500 as if it were a window through 
which the data plane 520 can be viewed, any particular 
portion of the data plane (for example, data subset 524) 
can be viewed. As will appreciated, in both the foregoing 
modes the speed of scrolling, the specific neutral tilt an- 
gle, and required pressures to initiate scrolling changes 
can be adjusted to fit a particular user. 

PORTABLE COMPUTER WITH HANDEDNESS 
DETECTION 

[0133] Pressure sensors have been added to aug- 
ment a conventional keyboard 551 enabled user inter- 
face to a hand holdable Windows ® CE class computer 
550 (i.e. a Cassio ® Cassiopia ®) schematically illus- 
trated in Figures 38 and 39. In this embodiment, user 
handedness was determined by using pressure sensors 
positioned on a right back-edge and a left back-edge of 
computer 550. User studies have found that the differ- 
ence in pressure between the right and left side gave a 
direct indication of handedness of a user. As respective- 
ly illustrated in Figures 38 and 39, handedness was 
used to justify formatted text 554 to the left (Figure 38) 
or right (Figure 39), thereby allowing more space 555 
on display 553 for an electronic annotation pen to be 
used to mark-up the text. 

[01 34] For both the embodiment of the invention illus- 
trated by Figures 35-37, and the foregoing embodiment 
illustrated in Figures 38 and 39, material deformation of 
a spongy, elastic, or otherwise deformable material 
must be measured. Although various techniques can be 
used to measure material deformation, including those 
based on imaging or fluid volumetric changes, one par- 
ticularly useful technique is based on the use of pres- 
sure transducers. Commercially available sensors 
measure pressure (indicative of material deformation) 
by converting a pressure change to a change in electri- 
cal characteristics. For example, inexpensive sensors 
that change resistance in response to pressure can be 
obtained in a variety of shapes and sizes, including pa- 
per thin sensors, and easily bendable sensor strips. 
Sensors of this kind can be customized to any particular 
shape or form that a gestural Ul might require. The 
change in resistance is usually linearly related to pres- 
sure, with the sensor generally being placed in a poten- 
tial divider network to model the pressure as a change 
in potential. For a practical circuit the resulting signal 
needs to be amplified, buffered and translated such that 
the change in value from minimum pressure to maxi- 



mum pressure spans a useful range. The modified sig- 
nal can now be fed into an analog to digital converter 
(ADC) to produce a digital representation of pressure. 
An 8-bit ADC can typically be used for most applications, 

5 however, if greater sensitivity to pressure changes is re- 
quired, a higher resolution ADC (e.g. 16-bit ADC) can 
be used. As will be appreciated, the ADC could be mem- 
ory mapped into the processor's address space as a pe- 
ripheral, or alternatively supplied as a retrofitted pres- 

io sure interlace to existing computers capable of benefit- 
ing from this system. Since an RS232 connection is an 
interface that is almost universally available on portable 
computers, one strategy is to have the parallel output of 
the ADC converted into a serial RS232 frame using a 

is serializer such as a UART and then level-shift and buffer 
the signal as specified by the RS232 standard. At the 
computer end of the serial interface, another level-shift- 
er and UART, the output of which is readable by the 
processor, performs the serial-to-parallel conversion. 

20 [01 35] As described with reference to Figure 40, in re- 
alizing a working system a microcontroller 564 (a Sig- 
netics 87c752) with an ADC built in to combine many of 
the I/O tasks in a single chip can be used in combination 
with level shifter 566 (a MAX3223). This approach has 

25 the advantage that intelligent processing of the input sig- 
nal is possible with software. Tilt measurement was pro- 
vided by a tilt sensor 567 connected to a buffer/amp 562 
to supply an analog signal to microcontroller 564. The 
pressure measurement can also be encoded within a 

30 protocol across the serial link. This particular microcon- 
troller 564 has five ADC inputs, but by making use of 
eight digital control lines, it is possible to use only one 
ADC input and one buffer amplifier 561 , to measure up 
to eight pressure points with pressure sensors 565. This 

35 is achieved by using the control lines to select only one 
sensor at a time and take a reading for each using a 
single input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory. The 
design is practical because the microcontroller can take 

^o measurements and make analog to digital conversions 
at a rate far higher than is needed to communicate with 
a host computer 569. 

[0136] For scrolling or handedness based software 
applications, 1 6 levels were determined to be adequate 

45 for representing a pressure measurement. In order to 
have a high data throughput to the host computer 569, 
each measurement was encoded in one byte of an 
RS232 frame such that the four lowest bits were the 
pressure representation and the highest four bits were 

50 the sensor ID. Each frame of the RS232 data was there- 
fore completely self contained. Of course any protocol 
that limits itself to a specific number of devices in its ad- 
dress space will eventually have the problem that some- 
day applications might be designed that need to refer- 

55 ence far more devices than are supported. The solution 
used in this protocol is to reserve the sensor ID number 
15 as a special value that can extend the semantics of 
the encoding to include an arbitrary number of bytes rep- 
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resenting a sensor or value. For the described software 
applications, the commonly used RS232 frame format 
(1 start, 8 data no parity, 1 stop bit at a baud rate of 9600) 
was chosen. 

[0137] In operation, the host computer 569 deter- 
mined handedness by utilizing information about the 
current pressure exerted on two subregions - one, on 
the back of the device, occupying roughly the left half, 
and the other, also on the back of the device, occupying 
the right half. The current pressure values were convert- 
ed from analog to digital form, such that a digital value 
of 0 (zero) represents no pressure, and a digital value 
of, for example, 15 represents maximum pressure. The 
detection circuitry then proceeds: 

if (left_sensor is high AND right_sensor is high) then 
conclude that the user is gripping the device 
with both hands 

else if (left_sensor is high AND right_sensor is low) 
then 

conclude that the user is gripping the device 
with the left hand only 

else if (left_sensor is low AND right_sensor is high) 
then 

conclude the user is gripping the device with 
the right hand only 

else if (left_sensor is low AND right_sensor is low) 
then 

conclude that the user is gripping the device 
with neither hand. 

[0138] Also, to optimize communication, pressure val- 
ues are only sent when they change. In order to account 
for jitter and error in the pressure sensors, a sensor is 
only considered "high" if its value is higher than some 
minimum threshold (e.g. '2' on the 0 to 15 range). 

SCANNER/PRINTER/COPIER SUPPORTING EDGE 
DEFORMABLE DISPLAY 

[0139] A scanner/printer/copier device 570 is sche- 
matically illustrated in Figure 41. As seen in Figure 41 
(and in more detail in Figure 42) the device 570 supports 
a display 574 having a deformable edge 572, with the 
combination roughly shaped like a piece of paper. In op- 
eration, a user can place a written document in the de- 
vice 570 for scanning. An electronic version of the 
scanned document is displayed (i.e. as text 575 in Fig- 
ure 42) on the display 574. By outwardly pulling the de- 
formable edge 572 as indicated by arrow 577, a user 
can instruct the device 570 to resize the document be- 
fore printing or copying. Pinching opposing sides of the 
deformable edge 572 (arrows 578) can further instruct 
the device 570 to print double sided copies. As will be 
appreciated, various other morphemes can be used to 
provide a simple interface for interacting with device 
570. 



TANGIBLE DEVICE FOR COMPUTER LOGON 

[0140] A password protected personal computer 612 
is a component of a computer system 610 that includes 

5 connection to a computer network 614. The tangible 
computing device takes the form of a stylized key 620, 
in accordance with a common metaphoric or analogical 
connection between access restricted physical security 
and a key. Like a conventional key used to open a door, 

io the stylized key 620 can "open" computer system 610 
by initiating logon procedures and providing passwords 
to a computer 612. 

[01 41] Typically, a wireless acoustic, radio or infrared 
system is used to permit intercommunication between 
is the key 620 and the computer 612. In Figure 43, a con- 
ventional infrared transceiver 618 connected to compu- 
ter 612 is used to receive and transmit infrared signals 
625 to an infrared transceiver 624 supported by key 620. 
The infrared transceiver 624 is connected to a proces- 
sor/memory unit 626 positioned within key 620. The key 
620 also includes processor/memory unit 626 connect- 
ed sensor 622 and feedback unit 628. The sensor 622 
can be constructed to detect tilt and rotation using con- 
ductive metal liquid switches, accelerometers, gravity or 
acceleration detecting microelectromechanical mecha- 
nisms, or any other convenient mechanism for detected 
transient tilts or rotations of the key 620. The sensor 622 
can also support pressure sensing mechanisms, 
squeeze or stretching sensors, thermal sensors, audi- 
tory sensors, or even complex bioidentification sensing 
units to determine fingerprints, voiceprints, or other 
unique identifiers of a user. The feedback unit 628 is 
used to generate user perceptible auditory, tactile, or 
visual feedback. This includes, for example, an audible 
"click", a vibrating element, a flashing light emitting di- 
ode, or any other suitable mechanism for indicating 
state change in the computer 612. 
[01 42] In operation, a user can log onto personal com- 
puter 612 by first bringing the stylized key 620 into a 
vicinity of the personal computer 612. The user simply 
rotates the key 620 (in the direction indicated by arrow 
627) to cause the tilt/rotation sensor unit 622 to report 
a rotation to the processor/memory unit 626. The proc- 
essor/memory unit 626 then sends (via infrared trans- 
ceiver 624) various user identification, logon, and pass- 
word information to computer 612 connected infrared 
transceiver 61 8. The computer 61 2 responds with an ac- 
knowledgement signal, causing the feedback unit 628 
to generate, for example, a harsh buzzing sound if the 
password is incorrect or, if the password is correct and 
logon successful, a series of brief audible clicks similar 
to the sound generated by turning tumblers in a conven- 
tional mechanical lock and key system. 
[0143] As those skilled in the art will appreciate, it is 
possible to substitute various other forms for the stylized 
key. A tangible computing device having the form of a 
garage door opener could be used, or any other suitable 
form sufficiently large to support the necessary proces- 
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sor, transceiver, and feedback electronics. In addition, 
various combinations of manipulation gestures such as 
previously discussed (e.g. squeezes, whacks, spins 
etc.) can be used as a substitute for a personal identifi- 
cation number (PIN). For example, activation of the key 
620 may first require three quick squeezes or other user 
definable manipulation prior to tilting for login. This 
would provide an additional measure of security in case 
of loss or theft of the key 620, just as requiring a PIN 
number reduces misuse of bank cards. 

ZOOMORPHIC DEVICE FOR COMPUTER LOGON 

[0144] Zoomorphic computer devices, discussed 
briefly in relation to Figure 1 , are a particularly preferred 
form of tangible devices. Small animal or humanoid 
shaped zoomorphic devices can help overcome some 
of the interface disadvantages associated with large, 
abstract, and/or imposing representations such as a hu- 
man head (e.g. as seen in connection with Figure 14), 
while still providing better recognizability than abstract 
shapes (e.g. cylinders or boxes). Like a tangible device, 
any zoomorphic device can represent either a user or a 
desired physical/virtual system, providing a correspond- 
ence in which the zoomorphic device can alternatively 
be known as an "avatar" or "talisman since it symbol- 
ically represents a user, a physical system, or a virtual 
system. 

[0145] For example, Figure 44 illustrates an alterna- 
tive computer logon/password system that uses one or 
more zoomorphic devices in place of the tangible styl- 
ized key of Figure 43. A "guard dog" 640 is a zoomorphic 
avatar formed to have internal electronics and transceiv- 
er 624 substantially identical to those discussed in con- 
nection with key 620 of Figure 43. Like the stylized key 
620, the guard dog can support pressure sensing mech- 
anisms, squeeze or stretching sensors, thermal sen- 
sors, auditory sensors, or complex bioidentification 
sensing units to determine fingerprints, voiceprints, or 
other unique identifiers of a user. The guard dog can 
also generate user perceptible auditory, tactile, or visual 
feedback to allow a user to determine various system 
states. Feedback includes, for example, canine audio 
utterances such as barks and growls, or electrically ac- 
tivated tail wagging. 

[0146] In operation, a user can log onto personal com- 
puter 612 by first bringing the guard dog 640 into a vi- 
cinity of the personal computer 61 2. After physically ac- 
tivating the guard dog 640 by a series of manipulations 
(e.g. squeezes 651 or rotations 653 that must be applied 
in a secret order), a user can place the guard dog at a 
predetermined position 655 on a position sensitive sur- 
face 646. Once properly activated and positioned with 
respect to the surface 646, the guard dog sends (via in- 
frared signal 625 generated by infrared transceiver 624) 
various user identification, logon, and password infor- 
mation to computer 612 connected infrared transceiver 
618. As previously discussed in connection with Figure 



43, the computer 612 responds with an acknowledge- 
ment signal, causing the guard dog 640 to generate, for 
example, a series of brief greeting barks or even elec- 
tromechanically activated tail wags (or movement of 

5 other appendages such as the guard dog's paw) if the 
password is correct and logon successful. When a 
guard dog is positioned incorrectly, such as illustrated 
with respect to guard dog 644 placed at position 657, 
the guard dog can emit a warning growl, rotate in place, 

10 move slightly to another place or position, or provide oth- 
er suitable indication that the logon procedure has 
failed. 

[0147] The use of secondary or auxiliary guard dogs 
that interact with other guard dogs or the personal com- 

15 puter 612 is also contemplated within the scope of the 
present invention. For example, a ■network" guard dog 
642, having substantially identical functionality and 
electronics as guard dog 640, can be used to logon to 
network computer system 614 after guard dog 640 has 

20 enabled logon to personal computer 612. Guard dog 
642 can communicate directly with the personal compu- 
ter 61 2, or via wireless link to the guard dog 640. As will 
be appreciated, guard dogs (or alternative tangible com- 
puting devices or zoomorphs) can convey various types 

25 of information to each other, to personal computers, or 
to networked computers. For example, guard dog 640 
can convey user profile information, address lists, orany 
other desired information to associated personal com- 
puters. In an alternative use, users can use their guard 

30 dogs to grant others having a suitable guard dog limited 
access to personal or network data files or applications. 

MEDIA SPACE PENGUIN 

35 [0148] As seen in Figure 45, a zoomorph shaped to 
resemble a penguin 720 having various accessories (e. 
g. LCD screen 722 for status display, hat 730, ear muf- 
flers 732, and darkened glasses 734) can be an inter- 
face to a media control system 710 using a wireless in- 
f rared transceiver 724. A media control system 7 1 0 (al- 
ternatively known as a "media space") includes a per- 
sonal computer 612, monitor 613, infrared transceiver 
618, and network system 614 substantially identical to 
that discussed previously in connection with Figures 43 

45 and 44. These interact with an audiovisual server 716 
connected through audio/visual switch 718 to various 
audiovisual input/output devices, including a camera 
740, microphone 742, and audio speaker system 744. 
The penguin 720 has various features or physical at- 

50 tributes that a user can symbolically or metaphorically 
relate to the media control system 710. The present in- 
vention takes advantage of some typical symbolic or 
metaphorical linkages to enable physical manipulation, 
orientation, or positioning of the penguin to be interpret- 

ss ed as commands to the media control system 71 0. 
[0149] For example, the penguin can be considered 
an avatar of the media space, and can be operated anal- 
ogous to the head previously described in connection 
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with Figure 1 4. For example, covering eyes of the pen- 
guin with the darkened glasses 734 means the media 
space camera 740 is turned off, covering the ears 
means the media space turns off or disables the micro- 
phone 742, closing the penguin's mouth (as indicated 
by arrows 727) mutes audio output from speaker system 
744. 

[0150] As those skilled in the art will appreciate, it is 
also possible to configure the penguin to be an avatar 
of a user. In such a case, the symbolic or metaphorical 
linkages are altered so that covering the eyes of penguin 
720 means the user does not want to see the media 
space as displayed by monitor 713, covering the ears 
of penguin 720 means the user does not want to hear 
the media space from speaker system 744, and cover- 
ing the mouth of penguin 720 means the user does not 
want to be heard, causing the microphone to be disa- 
bled. 

USER NOTIFICATION ZOOMORPH 

[01 51] In addition to acting as a user interface for con- 
trolling physical systems, a suitably modified zoomorph 
such as penguin can be used as for auxiliary or primary 
notification and messaging services. As seen in Figure 
46, the penguin 720 can be used to notily a user of re- 
ceipt of electronic mail (via network 614 to personal 
computer 612) by raising its arm 728 (in the direction of 
arrow 729) using conventional electromagnetic mecha- 
nisms built into the penguin 720. Advantageously, this 
allows a user to determine useful information (the pres- 
ence of waiting electronic mail) by the position of arm 
728 without explicitly accessing a computer or viewing 
the monitor. Alternatively, acoustic or light output can be 
used to signal receipt of electronic mail, alone or in ad- 
dition to physical movement of the zoomorph. A send- 
er's name, subject header, or even the mail message in 
whole or in part can optionally be displayed on the 
screen 722 of the penguin 720, as well as being availa- 
ble on the monitor 61 3. As will be appreciated, notifica- 
tion services are not limited to flagging receipt of elec- 
tronic mail, but can include notification or messaging for 
changes of state in a wide variety of physical and virtual 
systems. Such systems may include, but are not limited 
to people (a user himself or other people), environ- 
ments, computational systems, or automated tracking 
systems. Systems may be represented by one or more 
physical avatars, or alternatively, a single tangible de- 
vice or zoomorphic avatar may represent one or more 
physical or virtual systems. 

Claims 

1. A zoomorphic computer user interface comprising, 

a computer system directable both to accept 
user input to modify its system state, and to pro- 



vide user output, 

a zoomorphic unit connected to the computer 
system to provide user input and react to com- 
puter system output, 

5 a reference unit connected to the computer sys- 

tem and available for interaction with the 
zoomorphic unit, with positioning of the 
zoomorphic unit with respect to the reference 
unit directably modifying system state of the 

10 computer system. 

2. The zoomorphic computer user interface system of 
claim 1, wherein the zoomorphic unit further com- 
prises at least one sensor to monitor positional 

*5 changes within multiple subregions of the zoomor- 
phic unit. 

3. A zoomorphic computer user interface comprising, 

so a computer system providing input and output, 

a zoomorphic unit having at least one movable 
element, the zoomorphic unit being connected 
to the computer system, with the zoomorphic 
unit having a sensor mesh to detect multiple po- 
25 sitional changes within multiple subregions of 

the zoomorphic unit and provide input to the 
computer system, and a feedback unit for mod- 
ifying position of the movable element in re- 
sponse to computer system output. 

30 

4. A zoomorphic computer user interface according to 
claim 3, wherein the at least one movable element 
is an appendage. 

35 5. A zoomorphic computer for controlling a computer 
system providing input and output, the zoomorphic 
computer comprising, 

a zoomorphic shell 
40 a transceiver attached to the zoomorphic shell 

for communicating with the computer system, 
and 

a position detecting unit for determining posi- 
tion of the zoomorphic shell relative to the com- 
45 puter system, with change of position of the 

zoomorphic shell relative to the computer sys- 
tem changing state of at least one of the 
zoomorphic shell and the computer system. 

50 6. A zoomorphic computer according to claim 5, fur- 
ther comprising 

a movable element attached to the zoomorphic 
shell, and 

55 a feedback unit supported by the zoomorphic 

shell and maintainable in communication with 
the computer system, with the feedback unit 
modifying position of a movable element in re- 
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30 
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45 
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sponse to computer system output. 

7. A zoomorphic computer according to claim 5 or 
claim 6, further comprising a sensor mesh in contact 
with the zoomorphic shell, the sensor mesh detect- 
ing multiple positional changes within multiple sub- 
regions of the zoomorphic shell, and transmitting in- 
formation related to multiple positional changes 
within the multiple subregions to the transceiver to 
provide input to the computer system. 

8. A zoomorphic computer according to any of claims 
5 to 7, wherein position of the zoomorphic shell with 
respect to the computer system determines suc- 
cess of computer system logon. is 

9. A computer or interface according to any of the pre- 
ceding claims, further comprising an audio feed- 
back unit 

20 

1 0. A computer or interface according to any of the pre- 
ceding claims, further comprising a visual feedback 
unit. 
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